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ABSTRACT 

Empirical literature on two types of hearings, and 
their applicability in evaluating educational programs, is reviewed. 
An introduction describes the use and nature of hearings and 
distinguishes adversary hearings, in which two parties argue opposing 
sides of an issue, from committee hearings, in which a variety of 
positions are presented to a panel of decision-makers. Adversary 
hearings, the type primarily used in evaluation, are the main focus 
of the paper. Following a brief overview of literature on adversary 
hearings, the discussion covers strengths of the adversarial 
approach, hearing procedures (including a table illustrating basic 
steps and variations), applications (including a table showing recent 
uses of adversary hearings in evaluation), and problems and 
limitations of the approach. Committee hearings and their uses, for 
which much less literature is available, are then briefly examined. 
The paper concludes that (1) adversary hearings effectively involve 
large groups in complex issues, and (2) committee hearings "might 
provide some of the benefits of adversarial hearings without the same 
problems," and (3) "much more public trial, testing, and revision are 
needed" for both hearing types. (MCG) 
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PREFACE 



The Research on Evaluation Program is a Northwest Regional 
Educational Laboratory project of research, development, testing, 
and training designed to create new evaluation methodologies for 
use in education. This document is one of a series of papers and 
reporcs produced by program staff, visiting scholars, adjunct 
scholars, and project collaborators — all members of a cooperative 
network of colleagues working on the development of new 
me thodolog ies • 

Adversary and committee hearings have been advocated as procedures 
which can effectively involve large numbers of people in clarifying 
issues and examining human testimony in the evaluation of complex, 
highly politicized programs. This report reviews the strengths, 
procedures, and applications of these two methods over the last 
fifteen years, summarizing their problems and limitations as 
evidenced in field trials to date. An extensive bibliography is 
included. 

Nick L. Smith, Editor 
Paper and Report Series 
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ADVERSARY AND COMMITTEE HEARINGS AS 
EVALUATION METHODS 



Hearings are a common inquiry and information presentation 
device in our society. They are used in diverse applications 
such as to render judgments under the law, to review proposed 
legislation in Congress, to assess the impact of planned 
environmental changes, to make decisions about the release of 
individuals from prisons and mental hospitals, to provide public 
accountability for administrative actions, and for numerous other 
applications. 

In this report, I focus more narrowly on the use of hearings 
as a tool in conducting evaluations. Although the examples cited 
here concern primarily the evaluation of educational programs, 
the principles, procedures, problems, and benefits of the 
hearings process are probably applicable to a wider range of 
evaluation settings and topics. 

By "hearings, " I am referring to the structured, oral 
presentation of information, evidence, and argument in a group 
setting with representatives of the primary audience in 
attendance. There are many variations possible within this broad 
definition, however; for example, (1) the presentation may be 
highly structured and rule-governed (e.g., using a legal trial 
format) or more informal (e.g., a committee briefing); (2) the 
evidence and information presented may be anecdotal and 
testimonial (e.g., participants 1 perceptions) or formal and 
inquiry-based (e.g., from controlled studies or replicable 
analyses); (3) the purpose of the hearing may be simply to share 
descriptive information (e.g., presentation of alternative plans) 
or to arrive at a decision (e.g., to render a verdict or make a 
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final administrative determination); and (4) the audience may be 
a large public body (e.g., school district parents), a 
representative group (e.g., a grand jury), or an empowered 
official (e.g., a judge or an agency chief administrator). 

Although innumerable variations of the hearings process are 
possible for the sake of this review, I have defined two 
subcategories: 

(1) adversarial hearings , in which two opposing sides argue 
the merits of the case or problem at hand within a 
competitive and adversarial format. Civil and criminal 
trials are popular models for this type of hearing. 

(2) committee hearings , in which a number of positions (not 
just pro and con) are presented, often by representatives 
of a variety of interest groups. Congressional committee 
hearings, public policy hearings, presentations to boards 
of directors and advisory councils are models for these 
hearings. 

In practice, these distinctions are, of course, not always 
clear. Indeed, evaluation uses of hearings are marked more by 
their diversity than their similarity. To hazard an 
oversimplification, however, one might say that adversarial 
hearings tend to be more formalized and focused, with 
considerable attention to reaching a definitive conclusion. 
Committee hearings are more open to consideration of a range of 
alternative positions, with less focus on issue resolution and 
more on issue examination and information sharing. 

This report provides an introductory overview to the use of 
hearings in evaluation. I touch upon the strengths, procedures, 
and applications of evaluative hearings over the last fifteen 
years. Since my purpose is to provide a brief "state of the 
method 11 review, I have included an exhaustive identification of 
the literature and devote most attention to the problems and 
limitations of hearings as an evaluation method. This paper 
should provide those researchers considering the use of hearings 
with a solid, but concise, introduction. 
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By far the most applications of evaluative hearings have 
employed the adversarial approach, I therefore begin with that 
model and conclude with a summary of the few applications of the 
committee hearings approach. In both cases, ray focus is more on 
the actual uses of these methods than the theory of how they are 
supposed to work* 

Adversary Hearings 

Levine (1982) summarizes the adversarial hearings approach as 
follows : 

In essence, the adversarial model operates with the 
assumption that truth emerges from a hard, but fair 
fight, in which opposing sides, after agreeing upon the 
issues in contention, present evidence in support of 
each side. The fight is refereed by a neutral figure, 
and all the relevant evidence is weighed by a neutral 
person or body to arrive at a fair result* 

(Levine, 1982, 270) 

Early writings on the theory, use, and conduct of adversary 
hearings in evaluation included that of Owens (1973) , Wolf 
(1973), and Levine (1974). Subsequent writings have urged the 
expansion of adversary approaches (Wright and Sachse, 1977) , 
summarized case studies of adversary evaluations (Owens and 
Hiscox, 1977) , proposed criteria for judging the quality of 
adversary hearings (Worthen and Owens, 1978), and discussed 
adversary hearings as an example of one of several evaluation 
improvements possible by adapting procedures from the law (Owens 
and Owen, 1981; . (Additional writings have included Kourilsky 
and Baker (1976), West (1976), Shore (1977), Worthen and Rogers 
(1977) , Levine and Rosenberg (1979) 9 and Nadler and Shore (1979) . 

In passing, it should be noted that there have also been 
discussions and applications of adversarial procedures which have 
not employed the hearings process. For example, Levine (1973a) 
has suggested the assignment of a staff "adversary" to a research 
project to cross-examine each piece of evidence as it is 
collected. Kourilsky (1973a) described a procedure in which 
arguments for and against a particular program or policy are 



presented to a decision maker who questions the presenters and 
synthesizes a decision. Stake and Gjerde (1971) incorporated 
advocacy and adversarial statements in the final evaluation 
report of a summer institute. An advocate summarized the most 
favorable arguments in support of the institute, while the 
adversary summarized the most damaging criticisms. Criticisms of 
these approaches have also been made (cf. Kourilsky, 1973b; 
Levine, 1973b) . 

Strengths 

A number of strengths or benefits of conducting adversary 

hearings have been suggested in the literature, including that 

adversary hearings: 

permit the various persons affected by a program to be 

involved in its evaluation and thus incorporate a variety 
of perspectives (Wolf, 1975; Wolf, 1979); 

foster alternative interpretations of evidence prior to 
reaching a conclusion (Wolf, 1975); 

admit human testimony and judgment as evidence, in contrast to 
more traditional evaluation approaches (Wolf, 1975; Wolf, 
1979); 

present both pro and con evidence and provide for 

cross-examination of testimony (Owens et al., 1976); 

respond more directedly to audience information needs (Owens 
et al, 1976); 

incorporate the use of a wide variety of data while preserving 
the complexity of the program and its social setting 
(Wolf, 1975; Levine et al., 1979); 

No one suggests, of course, that adversary hearings are always 
the most desirable method. Worthen and Owens (1978), and Madaus 
(1982) suggest conditions under which adversary hearings are most 
appropriate; for example, tfhen the program is controversial and 
public opinion regarding its value is polarized, when decisions 
must be made about terminating a large program that affects many 
people, and so on. 

Experiences in actually applying adversary hearings tend to 
support these claims of the benefits of the process. 
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Procedures 

For detailed examples of the procedures for conducting 
adversary hearings, see the application reports listed below in 
Table 1. Wolf (1979) provides a generic structure for conducting 
adversary hearings, which includes: 

Issue generation , in which possible specific issues to be 
addressed in the hearing are identified and developed; 

Issue selection , in which issues not in dispute are set aside 
and those to be "tried" are prioritized and further 
developed; 

Argument preparation , in which the pro and con cases 

concerning the issues to be tried are developed through 
collection of evidence and the building and testing of 
arguments; 

Clarification forum , in which the actual hearing is held to 
present the pro and con cases, to evaluate the 
collective evidence and arguments, and to render 
decisions or recommendations. 

Many variations within this general structure are possible, 
some of which are outlined below. 



Basic Procedure 



Variations 



1. Issue (s) to be addressed 
are clarified and 
selected prior to hearing 

2. Two sides gather evidence 
to best make their own case; 
the evidence is collected, 
organized, and weighed, 
using interviews, observa- 
tions, document review, 
summaries of quantitative 
data 



Issues emerge as each side attempts to 
make its best case 



The same initial set of data is shaped 
by both sides who subsequently use it 
to argue pro or con 
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3. Persons affected by the 
program being studied are 
involved in the process 
of shaping issues and 
identifying evidence 

4. Hearing is held: 
evidence presented, 
witnesses testify and are 

. cross-examined 



Representatives of affected parties 
participate, or affected parties serve 
essentially only as audiences 



Some points may be conceded prior to 
hearing 

Expert testimony or documentary 

evidence may be admitted prior to the 
hearing 



5. Cases for and against Only case for or against is presented, 
are presented other side challenges arguments and 

evidence 

6. Jury reaches decision No decision reached, only information 

for decision provided 
No jury; general audience serves as 
jury 

No jury; judge (or decision-maker) 
makes decision 

Applications 

Probably the first formal application of an adversary hearing 
approach in educational evaluation was in 1970 (Owens, 1971) . 
Since then, there has been a variety of applications; adversary 
hearings have been used to evaluate school curricula, teacher 
training programs, graduate professional programs, and school 
policy in special education. Table 1 below contains a brief 
summary of nine formal applications of the adversary hearings 
procedure made to date. These applications range from nationally 
televised external studies to small-scale, in-house reviews. 
They evidence a variety of contexts, purposes, and procedures. 

Problems and Limitations 

Generic problems . Because hearings are a relatively new 
approach to the conduct of evaluation, considerable attention has 
been given to their limitations and to the problems of conducting 
them. Some of these difficulties have to do with the competitive 
nature of the adversarial process; for example, the qualifications 
and abilities of the adversaries may not be balanced (Wolf, 1975; 
Owens et al., 1976), the audience may be influenced more by the 
persuasiveness of the individual adversaries than by the evidence 
presented (Owens, 1973; Owens et al., 1976), or "winning" may 
become more the goal of the participants than clarifying and 
deciding issues (Levine et al., 1978; Wolf, 1979) — which may 
result in a distorted and unhelpful polarization of viewpoints 
(Worthen and Owens, 1978) . 
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Table 1 

Applications of Adversary Hearings 



Focus of 
Year Evaluation 

1970 Hearing process 
itself was 
primary focus 
of evaluation 



Issue (s) Addressed Highlights of Design Employed 



Should the curriculum 
"Man: A Course of 
Study" be adopted in 
the Hawaii public 
schools? 



Used an "administrative- 
adversary hearing" model 

Hearings officer , two 
adversaries 

Three audience representa- 
tives as judges 

Two-hour public hearing 

Ground rules explicated in 
advance 

Witnesses, charges f rebuttals f 
and redirect examination 



References 

Owens {1971) 
Owens (1973) 



1974 



1975 



1976 



1976 



Experience- 
based career 
ed. program 
for high school 
students 



University 
broad-based 
teacher educa- 
tion program 



Doctoral 
candidacy 
procedures in 
a clinical 
and community 
psychology 
program 



Statewide 
primary grade 
team teaching 
program (3 on 
2 Program) 



Should this program 
be adopted by your 
school? 



What impact has the 
new teacher education 
division had on the 
improvement of 
teacher preparation? 



Is the program, as 
operated , violating 
its stated standards 
and purposes? 



Is the program 
successfully meeting 
its goals? 



Video-taped hearing 
Witnesses for and against 

adoption 
Cross-examination of testimony 
External professors hired as 

adversaries 

After 6 months 1 preparation, a 
2-day , videotaped hearing was 
held before a 13-member 
national jury panel 

32 witnesses testified, testi- 
mony cross-examined, docu- 
ments entered as evidence 

Jury could ask questions, and 
rendered judgments on issues, 
including recommendations 
for program modification 

A hearings process was used to 
establish program purpose 
and standards 

A 1-1/2 day public hearing fol- 
lowed jury trial procedures 
closely except rules of admis- 
sable evidence were relaxed 

Jury had 4 hours to deliberate, 
reached near-unanimous deci- 
sions and program reform 
recommendations 

Extensive meta-evaluation of 
the study conducted 

Ten evaluators jointly designed 
study, data collection and 
analysis 

Two 4-person evaluator teams 
randomly assigned to pro or 
con positions; other 2 
evaluators served as arbiters/ 
facilitators 



Hiscox & Owens 

(1975) 
Owens et al. 

(1976) 



Arnstein (1975) 
Wolf (1975) 
Stenzel (1975) 
Bullock (1976) 



Garvey & Levine 

(1976) 
Levine (1976) 
Levine et al. 

(1978) 



Nafziger et al. 
(1977) 



(cont. ) 
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(Table 1 continued) 



Year 



Focus of 
Evaluation 



1977 



Public Law 
94-142: Educa- 
tion for all 
Handicapped 
Children Act 



1978 



How public 
schools are 
governed 



1981 



Minimum compe- 
tency testing 
programs in 
general 



1984 



Textbooks for 
an evaluation 
seminar 



Issue (s) Addressed Highlights o f Design Employed References 

Modified debate hearing 
procedure used in closed 
meeting with school officials 
and legislators 

One-hour , videotaped version 
shown on statewide television 
with audience input encouraged 



Will use of proce- 
dural documents be 
limiting? 
Will standardized 
procedures facilitate 
implementation? 
Will requirements 
increase quality of 
education? 



What are the policy 
alternatives for 
school governance in 
this district? 



What is MCT's effect 
on students? on cur- 
riculum and teaching? 
on public percep- 
tions of educational 
quality? 



Should last semes- 
ter's texts continue 
to be required for 
this semester? 



Three-day hearings held in 

each of 4 states 
Two teams organized in each 

state 

Several hundred persons inter- 
viewed in each state during 
5 rounds of interviews 

Written recommendations from 
hearings were used in final 
drafting of federal 
regulations 

Community-wide participation 

in process 
Hearing held over 3 consecutive 

evenings after several months 

of field interviews 
Recommendations from citizen 

panel presented to school 

board and in local media 

Pro and con teams given 10 
hours each to argue merits of* 
MCT in 3-day public hearing 

57 witnesses testified in 
federally sponsored hearing 
using nationally prominent 
scholars 

Videotaped by PBS, edited to 
3 hours, made available to 
local stations nationwide 

No jury or rendered decision/ 
viewing audiences to make 
"decisions" 

Adversaries were previous 

student users of the texts 
Judge and jury members were 

departmental colleagues of 

the instructor 
Teams had access to each 

other's data 
Jury could ask questions and 

had 30 minutes to render a 

verdict 



Wolf & Tyraitz 

(1977) 
Wolf (1979) 



Wolf (1978) 
Wolf (1979 



Herndon (1980) 
NIE (1981) 
Bourexis (1982) 
Drennan & 

Stalford 

(1982) 
Estes & Demaline 

(1982) 



Brown * Palmere 
(1984) 
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Other problems have to do with managing the hearing process 
itself. For example, adversarial hearings require extensive 
preparation time (Owens, 1973). Cost estimates from Levine et 
al. (1978) sugqest that only 16 percent of their total effort 
went into the hearing itsel*, over 80 percent went into 
preparation, case building, and managing the process. Further, 
time limitations can force a focus on only a few issues with 
important evidence being omitted from the proceedings (Wolf, 
1975; Owens et al., 1976, Levine et al., 1978). 

Lack of experience with the hearing process also creates 
problems, it is difficult to clearly present complex 
quantitative data in a hearing format (Levine et al, 1978), and 
issue framing, selection, and presentation is a most 
time-consuming and troublesome task (Arnstein, 1975; Levine et 
al. , 1978; Wolf, 1979). (I will return to this problem in a 
moment.) 

Finally, few individuals are adequately prepared to 
participate in adversarial evaluations. It is difficult for 
citizen panels to produce recommendations that are specific, 
operational, and helpful. The issues, evidence, and arguments 
presented to them must be detailed and specific (Wolf, 1979) . It 
is also difficult to identify persons with proper skills and 
characteristics to function effectively in roles of panel member, 
adversary team leader, proceedings moderator, etc. Experience 
gained from the studies listed in Table 1 does suggest, however, 
that clear charges, explicit instructions, shared expectations, 
and some training are effective in improving the quality and 
impact of the proceedings (Wolf, 1975; Levine, 1978; Wolf, 1979; 
Braithwaite, 1980) • 

Specific studies . The 1976 study of a statewide team 
teaching program— the Hawaii 3 on 2 Program (Nafziger et al., 
1977) , and the 1981 hearings on minimum competency testing — the 
MCT Clarifications Hearings (Herndon, 1980; Drennan and Stalford, 
1982; Estes and Demaline/ 1982) have been the most widely 
publicized and criticized applications of adversary hearings 
procedures to date. Brief summaries of the criticisms of each 
follow. 
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The Hawaii 3 on 2 Program was a statewide team- teaching 
program in which t s ree teachers worked with two classrooms. Two 
of the evaluators participating in the adversary process (ro? 
Table 1), James Popham and Dale Carlson, subsequently published a 
highly critical assessment of the method based on their 
experiences (1977). Of their six major criticisms, three were 
similar to problems previously identified: the imbalance in the 
skills of the adversaries, the difficulty in framing issues, and 
the the expensiveness of the process. In addition, they charged 
that too much confidence was being placed in the power of the 
adversarial process, that the intent of a fair, unbiased hearing 
process could be subverted by an unscrupulous decision maker, and 
that fallible judges or hearing moderators could severely damage 
the outcome of the proceedings. In response, Jackson (1977) 
criticized Popham and Carlson for failing to mention that their 
criticisms were equally applicable to other available evaluation 
procedures, for ignoring the possible benefits of the adversary 
approach, and for reaching a premature judgment based on limited 
experience. Thurston (1978) similarly charged that the Popham 
and Carlson criticisms were irrelevant, unfair, and failed to 
adequately address two serious shortcomings in the model, the 
definition of issues and the use of the jury. 

Even more has been written about the minimum competencies 
testing clarification process, including papers on its 
procedures, results, and evaluation (Herndon, 1980; NIE, 1981; 
Bourexis, 1982; Drennan and Stalford, 1982; Estes and Demaline, 
1982). In addition, the substantive content of the hearings 
(Madaus, 1981b; Popham, 1981a; Shoemaker, 1981; Thurston and 
House, 1981) and the role of television in the evaluation 
(Herndon and Shoemaker, 1981; Shoemaker, 1982) have been 
discussed. 

The leaders of the pro and con adversary teams of the MCT 
evaluation, James Popham and George Madaus, respectively, wrote 
critiques of the clarification process. Madaus (1982) reported 
that although the sponsoring agency, NIE, and the public were 
apparently pleased with the hearing process, that he would not 
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choose to repeat the experience. He cited such problems as the 
way the use of public television affected the proceedings (e.g., 
uafc oeiecciou of witnesses, tne presentation ot complex evidence, 
and the editing of 24 hours of material down to 3 hours) , 
inadequate budget, insufficient hearing time for testimony, the 
labeling of teams as "pro" and "con," the intrusiveness of the 
sponsoring agency, and the difficulties of sharing data across 
the teams. Popham (1982) complained about the problems of team 
competitiveness in spite of attempts to be collaborative, the 
effect of the use of public television, and the high expense of 
the process. Popham concluded that he was more positive about 
adversary hearings after this study than after the 3 on 2 
evaluation (e.g., Popham and Carlson, 1977), and that he would 
welcome the opportunity to repeat the experience. Madaus (1982) 
concluded that adversary hearings of this nature were probably 
better suited to state and local rather than national settings 
and that a documentary format would have done a better job of 
clarifying issues for television broadcast than an adversary 
hearing. 

Issues clarification . One of the most difficult aspects of 
the adversarial hearings process seems to be the identification, 
selection, and presentation of issues for trial and resolution. 
Allowing issues to emerge as each side attempts to present its 
own best case is not only inefficient, but confuses participants, 
inhibits the presentation of clear, relevant evidence, and makes 
it difficult for the jury or panel to reach specific, useful 
recommendations and decisions. 

A number of approaches to issue clarification have been 
tried. Wolf (1975) used a committee to review program complaints 
and to select issues to be tried. Levine et al. (1978) employed 
a " legislative hearing 11 process, prior to the actual trial, to 
establish a statement of program purpose and standards against 
which the program's performance could be tried. Thurston (1978) 
has suggested using a non-jury legal approach such as an 
appellate court model to better formulate issues. Under the 
appellate model, the educational court would precisely and 
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narrowly state what is at issue and then decide the issue based 
o* standard educational practice and the evidence at hand, 
perhaps other approaches such as Dillon's (1934) procedures for 
question formation could be applied to the identification and 
shaping of issues for adversarial hearings. 

As a result of their experiences with the MCT clarification 
hearings, both Pophara (1981b) and Madaus (1981a) have offered 
suggestions on how better to frame issues on adversary hearings 
of MCT programs: Pophara arguing that issues ought to be phrased 
in terms of program effects, Madaus arguing that they ought to be 
in terms of program functions. House et al. (1984) pay 
considerable attention to the problem of issue specification in 
their analysis of the MCT process. They suggest that the vague 
specification of abstract issues was a primary cause of the 
confusion and lack of focused argument in the proceedings. In 
concert with most of the writers who have participated in 
adversary hearings, they strongly emphasize the necessity for a 
clear delineation of the contested issues which are then joined 
in argument by the adversary teams. 

Summary 

As Levine (1982) points out, the studies of adversary 
evaluation presented in the literature say little about how the 
jury or panel's verdict subsequently influenced the program under 
study. Much of the attention to date has focused on how to make 
the process work. The participants, and especially the 
organizers of these studies, report general satisfaction with 
adversarial hearings. While they .acknowledge a number of 
problems (especially the considerable effort involved, the 
difficulty with issue clarification, and problems of the 
competitive nature of the process) , they remain convinced that 
the procedure is an effective way to provide for public 
participation and scrutiny of human testimony in evaluating 
complex programs and policies. Levine 's (1982) comments seem to 
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reflect the position of most advocates of the adversary hearing 
process: 

The need to develop fair methods of fact-finding and 
dispute resolution, and convincing methods of presenting 
controversial findings is as important as the need to 
develop more powerful or efficient research methods, 

(Levine, 1982, 276-277,) 

Committee Hearings 

There is much less writinc and experience available on the 
use of committee hearings as an evaluation method* Levine et al. 
(1978) did employ a "legislative" hearing approach to clarify 
program purposes and standards prior to an adversarial hearing. 
They report that some participants felt that the purpose of the 
evaluation could have been met solely with the legislative 
hearing in which there were no clearcut sides and no teams acting 
in adversarial roles. Most witnesses felt more comfortable in 
the legislative hearing where they were allowed to give fuller 
versions of their own stories, Thurston (1978) has also 
suggested that administrative hearings (a variation of committee 
hearings) is an alternative model without certain problems of 
adversary hearings, Stenzel (1982a, 1982b) has written the most 
formal descriptions of the use of committee hearings in 
evaluation. His descriptions have been developed by using 
congressional committee hearings as an evaluation model. 

Committee hearings (and variations based on administrative, 
legislative, or congressional hearing models) have much in common 
with the adversary hearings just reviewed. They consist of 
public hearings where a variety of persons affected by the 
program being evaluated may observe, provide testimony, and ask 
questions. Important differences are that committee hearings are 
not designed as adversarial procedures and a variety of positions 
are explored, not just pro and con. More emphasis is placed on 
issue clarification and information sharing and less on rendering 
a verdict, although recommendations and decisions might result 
from the deliberations, 
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The general procedures for conducting committee hearings are 



similar to those for adversary hearings: 



parent body establishes a committee and chooses a 

chairperson: 
council and professional staff are appointed; 
professional staff clarify committee charges and 

issues, conduct research, and select witnesses; 
a hearing is held, moderated by the committee 

chairperson, with assistance from the counsel 

and staff; 

opening statements and witness testimony are 
given; 

committee members examine evidence and question 
witnesses; 

professional staff prepare a preliminary report; 
committee finalizes report and submits it to its 
parent body. 



As with adversary hearings, many variations of this basic 
procedure are possible. For example, Stake and Balk's (1982) use 
of a hearing process to brief an ad hoc panel can be seen as a 
variation of a committee hearing. See Stenzel (1982a) and St. 
John (1984) for detailed dicussions of procedures and role 
responsibilities of participants. 

There have been two clear applications of committee hearings 
as an evaluation technique. 

In 1975, the Illinois Office of Education established a 
9-person examination committee to review the validity and 
recommendations of a commissioned evaluation. Third-party 
evaluators had been hired to evaluate 13 regional projects 
that provided services to low- incidence handicapped 
students in Illinois. The evaluators presented the report 
and findings to the committee which included 
representatives of private schools, public special 
education cooperatives, parent groups, and state office 
personnel. The committee examined the report, heard 
testimony, questioned the witnesses, and prepared a 
written report to the Office of Education (Illinois Office 
of Education, 1975; Jones, 1976; Stenzel, 1982a). 

A textbook selection committee in an Illinois school 
district used the committee hearing process to select a 
history book for adoption. As the parent body, they 
appointed a committee of 5 history teachers who conducted 
a preliminary investigation (needs analysis, reading 
analysis, and pilot studies) of prospective texts. Three 
textbooks were chosen and a hearing was held at which 
sales representatives, teachers using the texts and their 
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students, and a reading consultant provided testimony. 
Committee members questioned the witnesses and recommended 
a single text for district adoption (Stenzel, 1982a) 

Just as committee hearings share some of the advantages of 
adversary hearings (open participation of affected parties, 
cross-examination of evidence) , they also share some of the same 
problems; they are expensive, time consuming, and include tasks 
for which participants have little training or experience. 
Unfortunately, there has been so little formal experience with 
committee hearings in evaluation to date, that we know little 
about the true strengths and limitations of this method as it 
might actually be applied. While experience with the adversary 
hearings process suggests possible benefits and problems of 
committee hearings, actual experience with the method is 
currently very limited. 



Although the history of adversary and committee hearings in 
evaluation has been relatively brief (15 years), several 
applications have been highly public and their discussion 
lively. Much attention has been devoted to the problems of 
actually conducting adversary hearings, which is understandable, 
given their dramatic departure from the more traditional forms of 
evaluation. Experience with adversary hearings has highlighted 
problems of issue clarification and the costliness and 
competitive nature of the process. Although the number of 
applications is not large, advocates and critics of adversary 
hearings still seem to agree that the process is an effective way 
to involve large groups of people in the collection and 
examination of testimony and argument about complex, highly 
political programs. Committee hearings, a largely untried method 
in evaluation, might provide some of the benefits of adversary 
hearings without the same problems. For both methods, much more 
public trial, testing, and revision are needed before final 
judgments of their utility will be warranted. 



Conclusion 
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